Fixed pattern noise removal in CMOS imagers across various operational conditions

ABSTRACT

A method of minimizing noise in an image produced by an electronic imager comprising: determining a correction system for a range of imager integration times and a range of imager temperatures for an electronic imager which has taken a series of dark capture images and a series of flat field capture images in a calibration mode; and applying the correction system to an image produced by the electronic imager in an image capture mode.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application is a Continuation-In-Part application of U.S.patent application Ser. No. 10/242,865, filed Sep. 13, 2002.

FIELD OF THE INVENTION

[0002] This invention applies generally to the field of electronicphotography and relates in particular to providing smoothed digitalimages with reduced noise.

BACKGROUND OF THE INVENTION

[0003] In electronic photography, CMOS imagers can possess higher levelsof imager noise than their predecessors CCD imagers. This noise can beof the form of temporal variation and fixed pattern noise. Fixed patternnoise includes Dark Fixed Pattern Noise, which is the pixel to pixelvariation in response offset and Pixel Response Non-Uniformity which isthe pixel to pixel variation in response to a given exposure.

[0004] Noise reduction is practiced in the art using dark fixed patternsubtraction as in U.S. Pat. No. 6,424,375. Here an electronic circuit isused to remove dark fixed pattern noise by electronically adjustingpixel responses to align them to an aim response. U.S. Pat. No.6,418,241 discloses a system in which column biases are corrected aftermeasuring the average of each column and adjusting each column to someaim bias. The Canon D30 digital camera also apparently performs darkcaptures with the shutter closed in order to obtain an estimate of thesensor's dark frame response. Also proposed is to use a dark framecapture and a flat field capture before every image capture. This is apractical impossibility in typical picture taking. There is thus a needfor improved technique for minimizing fixed pattern noise.

SUMMARY OF THE INVENTION

[0005] According to the present invention, there is provided a solutionto the problems of the prior art.

[0006] According to a feature of the present invention, there isprovided a method by which noise can be removed from a digital imagecaptured with an image sensor operating over a wide range ofenvironmental and operational conditions. This method removes both darkfixed pattern noise and pixel response non-uniformity which vary as afunction of imager temperature and imager integration time.

ADVANTAGEOUS EFFECT OF THE INVENTION

[0007] The invention has the following advantages.

[0008] 1. Dark fixed pattern noise correction is optimized to correctdark fixed pattern noise associated with imager integration time andimager temperature.

[0009] 2. Pixel response non-uniformity correction is optimized for arange of operational conditions of imager integration time and imagertemperature.

[0010] 3. In an alternate embodiment, the mean of dark pixels of theimager captured concurrent with the image are used to adjust both darkfixed pattern noise and the pixel response uniformity.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011]FIGS. 1 and 2 are block diagrams of embodiments of the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0012] Image Sensors can produce an undesirable response known as fixedpattern noise (FPN). FPN is defined as a non-image pattern produced byan image sensor that does not change from capture to capture. Temporalnoise, by contrast does change from capture to capture and is random.The present invention addresses FPN. There are two types of FPN, darkfixed pattern noise and pixel response nonuniformity (PRNU). Dark FPN isdescribed as pixel to pixel offset in response when operating the imagerin the absence of incident illumination (e.g., the shutter is closed orthe lens is capped). PRNU is described as the response of an alreadydark FPN corrected imager, to uniform incident illumination (e.g.,imager imaging an integrating sphere). The present invention effects thecorrection of both of these types of noise (Dark FPN, PRNU) across arange of operational conditions (imager temperature and imagerintegration time).

[0013] While the image noise the invention reduces is referred to asfixed pattern noise, it does vary with the operational conditions ofimager integration time and temperature. This variation is however quitepredictable, where temporal noise is highly variable from capture tocapture and unpredictable.

[0014] The strategy or procedure followed in this invention is toperform a calibration in which the imager is temperature controlledacross the range of temperatures the imager will experience in use, andfor each temperature, the imager is also operated across the full rangeof imager integration times. One can envision a 2D matrix of time andtemperature: 1/250 1/125 1/60 1/30 1/15 1/8 1/4 1/2 sec sec sec sec secsec sec sec 1 sec 2 sec 4 sec 8 sec  0 deg C.  5 deg C. 10 deg C. 15 degC. 20 deg C. 25 deg C. 30 deg C. 35 deg C. 40 deg C. 45 deg C. 50 deg C.55 deg C. 60 deg C.

[0015] In each cell of this matrix, a series of dark captures (e.g.,lens capped) are taken and a series of flat fields (e.g., integratingsphere) are captured. The dark captures are averaged together in orderto remove temporal noise from the estimate of the dark fixed patternnoise at that time and temperature. The flat field captures aresimilarly averaged together, but only after they have been eachindividually corrected for dark FPN. This is achieved by subtracting thedark FPN estimate computed above (by averaging together each darkframe), from each individual flat field captures. After each individualflat field image has been dark corrected, they can all be averagedtogether. This frame averaged flat field image shows any pixel responsevariation residual in the image. Each color channel is then used to forman aim response to which all pixels will be gained. The aim response isdefined as the average of each color channel in the center ¼ of theimager's format. After the 3 aim values are defined (e.g., RGB), a perpixel gain is computed.

[0016] That gain is defined as:

Gain_(ij)=Aim_(Red)/PixRsp if a Red Pixel

Gain_(ij)=Aim_(Green)/PixRsp if a Green Pixel

Gain_(ij)=Aim_(Blue)/PixRsp if a Blue Pixel

[0017] This invention uses one or more dark frames in a calibrationphase to estimate a dark current generation rate for each pixel, thenuse this model to calculate a dark frame to be subtracted duringprocessing for a particular scene.

[0018] One model is:${{CV}_{D}( {r,c} )} = {{{Gs}( {r,c} )}\frac{At}{q}^{{- {({E_{G} - E_{T}})}}/{kT}}}$

[0019] In this equation:

[0020] CV_(D)(r,c) is counts of dark current signal, for each pixel(row, column index)

[0021] G is the analog gain level

[0022] s(r,c) is a scaling factor for current generation for each pixel

[0023] A is pixel area, q is the charge an electron, E_(G) is the bandgap, E_(T) is the impurity energy gap, t is integration time, k isBoltzmann's constant, and T is temperature in Kelvin. By acquiring oneor more dark frames at a calibration time, we can estimate:${s( {r,c} )} = {{{CV}_{DC}( {r,c} )}\frac{q}{G_{C}{At}_{C}}^{{({E_{G} - E_{T}})}/{kT}_{C}}}$

[0024] In this equation, CV_(DC)(r,c) is a mean dark frame atcalibration time, G_(C) is the analog gain at calibration time, t_(C) isthe integration time for the calibration frames, and T_(C) is the sensortemperature for the calibration frames. This estimation is actuallyoversimplified, because an actual dark frame has our usual pattern, lagand nonlinearity artifacts. Thus, CV_(DC)(r,c) is really a dark frameafter having lag, column offset, column gain, and linearity correctionapplied.

[0025] After both dark FPN and PRNU correction maps are defined per cellin the matrix above (across operational time and temperature), thefunctional relationship between the independent variables of time andtemp and measured dark FPN and PRNU maps is assessed. The imager'snominal (nominal or typical operational conditions) FPN, at a minimumwill be scaled and or biased as a function of integration time andtemperature when the imager is operating in other than nominaloperational conditions. That functional relationship is determined withregression. The regressions are linear, higher order or an exponentialfunction in time and temperature.

[0026] The estimates of both dark FPN and PRNU correction images areimproved estimates relative to performing a dark field capture and aflat field captures before every image capture, since the inventionprovides multiple frame averaging at calibration time, thus removing anytemporal noise from these map estimates. An alternative embodiment ofthe invention is to make the biasing and scaling functions dependent ononly the mean dark response taken from the imager's dark pixels, at timeof scene capture.

[0027] Referring to FIG. 1, there is shown an embodiment of theinvention. As shown, a digital camera includes an image sensor 10 whichproduces an analog image of a scene and analog to digital converter(A/D) 20 which converts the analog image to a digital image 30.According to the invention as discussed above, exposure (integration)time and imager temperature corrections 60 and 70 are used to correctthe digital image 30 for fixed pattern noise to produce a correctedimage 50.

[0028]FIG. 2 shows an alternative embodiment wherein fixed pattern noisecorrection 40 is dependent on the mean dark response taken from thesensor 10 dark pixels 72 at the time of the scene capture.

[0029] The invention has been described in detail with particularreference to certain preferred embodiments thereof, but it will beunderstood that variations and modifications can be effected within thespirit and scope of the invention.

PARTS LIST

[0030]10 sensor

[0031]20 analog to digital converter

[0032]30 digital Image

[0033]40 FPN correction

[0034]50 corrected image

[0035]60 exposure time

[0036]70 imager temperature

[0037]72 dark pixel statistics

What is claimed is:
 1. A method of minimizing noise in an image producedby an electronic imager comprising: determining a correction system fora range of imager integration times and a range of imager temperaturesfor an electronic imager which has taken a series of dark capture imagesand a series of flat field capture images in a calibration mode; andapplying said correction system to an image produced by said electronicimager in an image capture mode.
 2. The method of claim 1 wherein saiddetermining a correction system includes determining a single fixedpattern noise (FPN) dark map and a single FPN pixel responsenon-uniformity (PRNU) map based on said series of dark capture imagesand said series of flat field capture images.
 3. The method of claim 1wherein said determining a correction system includes taking a series ofdark capture images and a series of flat field images for eachtemperature and imager integration time of a matrix of a plurality oftemperatures and imager integration times, averaging the dark captureimages for each said temperature and imager integration time to removetemporal noise, connecting each flat field image by subtracting saidaveraged dark capture image, and averaging said flat field images toproduce a pixel response non-uniformity correction.
 4. The method ofclaim 1 wherein said electronic imager has three different colorchannels and wherein said correction system effects an aim response ineach said color channel to which all pixels will be gained and per pixelgain is then computed.
 5. The method of claim 4 wherein said three colorchannels are R(Red), (G)Green, (B)Blue color channels.
 6. The method ofclaim 4 wherein said aim response is defined as the average of eachcolor channel in the center region of the imager's format.
 7. The methodof claim 6 wherein said center region is ¼ of the center of the imager'sformat.
 8. A method of minimizing noise in an image produced by anelectronic imager having image capture pixels and dark pixelscomprising: capturing an image with said image capture pixels; andcorrecting said captured image as a function of the mean dark responsetaken from said imager's dark pixels at the time of said image capturing